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Abstract 



This article details the development and psychometric analysis of an instrument used 
for workshop evaluation. The instrument was designed to assess the effectiveness of the 
workshops, the instructors conducting the workshop training, and the training materials. 
The instrument used a Likert agree/disagree format. A psychometric analysis of scores 
from n=584 participants found two factors that validated the proposed structure of the 
instrument. The first factor pertained to the workshops and training materials, while the 
second factor pertained to the instructors themselves. The Cronbach alpha value for the 
instrument was 0.88. The authors also made recommendations for generic use of the 
instrument in various training and assessment environments. 
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Development of a Workshop Evaluation Instrument 



Introduction 

In today’s instructional and training environment, most corporations are interested in 
how effective their products and services are in their specific business or industry. 
Companies spend millions of dollars every year conducting market research to determine 
customer perceptions and trends. This document is an analysis of one such effort to 
obtain pertinent, accurate information about how effective a specific training service was 
to a very specific clientele. 

In the late 1990s, a Texas-based computer software company began conducting 
software training workshops for insurance adjusters. The company management felt that 
a training satisfaction survey instrument would help determine the effectiveness of the 
workshops, the instructors conducting the workshop training, and the training materials. 
Therefore, the authors proceeded to develop a workshop evaluation instrument. 

The instrument was used first for a one-day workshop. The workshop was later 
expanded to a two-day class and then to a three-day class. During this time, the 
instrument questions stayed the same. However, the authors changed the layout design 
and font type after the instrument was first used. 

As previously mentioned, the purpose of the instrument was to obtain workshop 
participants’ opinions on such specific items as the content knowledge of the instructor, 
the quality of the workshop materials, and the overall effectiveness of the training 
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Method 

Instrument Design 

The design of the instrument was important, and many factors had to be considered. 

A Likert type scale was adopted for the instrument to determine the degree of satisfaction 
or dissatisfaction with each question. The decision was made to use the Likert scale with 
the descriptors “Strongly Agree”[5], “Agree”[4], “Neutral”[3], “Disagree”[2[, and 
“Strongly Disagree”[l] for the purpose of determining the participants’ perceptions and 
opinions about training. In an agree/disagree context, Jenkins and Taber (1977) found 
that the number of response categories above five did not, in any situation, yield a 
significant increase in Likert discriminability. Neumann and Neumann (1981) concluded 
from their research that the 5-point scale appeared to be the most convenient to use in 
attitudinal survey. McKelvie (1978) proposed using five or six categories. He further 
suggested there was no psychometric advantage in a large number of scale categories, 
and that discriminative power and validity could be reduced when fewer than five 
categories were used. The design of the scale focused on the questions being easy to 
understand and the participants being able to complete the questions in a relatively short 
time. The instrument was administered at the end of the final day of training. The 
instrument items follow in their sequential order from item one through item ten: 

(1) The tr aining room was comfortable, (2) The training materials were easy to follow 
and understand, (3) The workshop has helped me understand the information flow within 
the software, (4) I can now write estimates on the DDS Multi-line Pro, (5) The 
instructor(s) was well organized, (6) The instructor(s) was friendly and easy to listen to, 
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(7) The instructor(s) answered questions quickly and knowledgeably, (9) Overall, the 
instructor(s) did an excellent job, and (10) Overall this was an excellent workshop. See 
the document appendix for a copy of the Workshop Evaluation Sheet. 

Instrument questions also assessed the date of the class, the location of the class and 
the instructor’s name. An optional comments section was provided at the bottom of the 
ins trument to allow the participants to note any comments they wished to add. The 
comments section was labeled with a sentence stating: “Please share any comments that 
will help us improve the quality of our training sessions. (We Really Want To Know!)” 
The wording of this sentence encouraged constructive criticism and had the phrase, “We 
Really Want To Know!” The intent was to let the participants know that the instructors 
were serious about listening to the participants’ comments. 

Participants 

The sample size for the study was n — 584 participants. The typical participant at the 
workshops was an insurance property adjuster or an independent insurance property 
adjuster. These were individuals who worked on a contract basis with insurance 
companies or independent adjusting companies to write estimates on damage to 
residential and commercial properties. The majority of these individuals fell into the 
category of catastrophe adjuster, meaning they worked property claims where the damage 
was caused by natural or man-made disasters such as hurricanes, hail and wind storms. 
This profession usually attracts individuals who have a strong construction background in 
order to understand structures and structural damage. The average independent adjuster, 
as well as those in the training sessions, was someone typically age 40 or older. This is 
because of the tremendous amount of time away from home that the job requires (some 
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storm duties may last over a year). It is also a profession that requires long hours of 
work. Adjusters assess property damage during the day and write estimates, reports, and 
call insureds in the evening. 

Procedure 

As is standard with most evaluative participant surveys for seminars or workshops, the 
instrument was given at the end of the last day of instruction. Generally the session 
instructor pre-filled the date of the workshop, the instructor’s name, and the class 
location. When first utilized, the instrument was handed out after all the training was 
completed. The problem with this method of distribution was that most individuals were 
in the process of packing up their belongings and preparing to leave the class. Because of 
this some of the surveys were filled out very hurriedly, and there were very few added 
comments. This caused some concern about the accuracy of the collected data; therefore, 
the instrument was added at the end of the training workbook, and the participants were 
told about the evaluation instrument during the class orientation. On the last day of class, 
immediately after the last exercise, the participants were asked to turn to that page of 
their workbooks. They were asked to complete the survey, remove the survey page, and 
leave it at their workstations. This method proved to be much more effective than the 
previous distribution method, and the responses seemed to be much more thought out. It 
was also noted that the comments section of the survey seemed to be utilized much more 
with this method of application. The input that was received was consequently very 
helpful in fine-tuning the training sessions themselves. 
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Results 

Data Analysis 

The authors used the SPSS principal components program (SPSS, Inc., 2001) to 
examine the factorial validity of the instrument. Nunnally (1967) noted that some 
researchers refer to construct validity as “factorial validity.” Furthermore, factor analysis 
is an attractive method for evaluating validity because it focuses on the reliable 
components of test data (Gorsuch, 1983). Thompson (1989) noted that the “common 
variance” represented by indices of association tended to represent reliable variance, and 
since it is from these indices that factors are extracted, it follows that factors tend to be 
constructed from the “true score” components of variables. 

A fundam ental question that has to be addressed is whether different solutions will 
emerge when different factoring procedures are used. This question can be rephrased to 
ask when performing a principal components analysis versus a principal factor analysis 
whether different factors will emerge if the researchers put 1 .00s in the main diagonal 
rather than communalities. It should be noted that the number of variables in the analysis 
itself affects the degree of difference between the two analyses. That is, the proportion of 
entries involving the diagonal of the correlation matrix becomes increasingly smaller 
with larger variable sets. For example, with five variables, 20% (5/25) of the entries 
involve the diagonal; however, with 10 variables, only 10% (10/100) of the entries 
involve the diagonal. Gorsuch (1983) stated that when there were a large number of 
variables having moderate communalities, the differences were negligible. Cliff (1987), 
Diekhoff (1992), and Velicer and Jackson (1990) noted that researchers seldom see 
substantial differences using these two different procedures. 
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Determining the number of factors to extract from the correlation matrix is a 
fundamental decision in any analysis (Thompson & Borrello, 1986). Most researchers 
follow the recommendations of Guttman (1954) and extract all factors with eigenvalues 
greater than one. The criterion has been shown to be quite accurate when the number of 
participants is greater than 250 and when the mean communality is greater than or equal 
to 0.60 (Stevens, 1986). In this study, there were about 600 participants, and the mean 
communality was 0.65. Consequently, all principal components with eigenvalues greater 
than one were extracted and rotated using varimax criterion. Individual items were 
retained if they had a factor structure coefficient greater than or equal to 1 0.50 1. The 
first principal component yielded two components. The prerotation eigenvalues for the 
components were 5.04 and 1.47. The first rotated component accounted for 37.5% of the 
variance, while the second rotated component accounted for 27.5% of the variance. The 
cumulative explained variable was 65.0%. See Table 1 for the descriptive statistics for 
the instrument items. Table 2 presents the correlation matrix for the items, and Table 3 
presents the rotated principal components matrix. The Kaiser-Meyer-Olkin (KMO) 
measure of sampling adequacy was 0.89, which was greater than the 0.5 value needed for 
a satisfactory factor analysis to proceed. A Bartlett’s test of sphericity was performed to 
test the null hypothesis that the original correlation matrix was an identity matrix. The 
value 3057.83 was significant at the 0.000 level. Therefore, it was concluded that the 
correlation matrix was not an identity matrix and that it was appropriate to use factor 
analysis as a multivariate analytical technique. 

In addition to using component analysis to assess the appropriate conceptual variables 
and instrument dimensions, in psychometry there is also the need to assess the reliability 
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of the instrument scores. Cronbach (1970) cited .50 as the minimum reliability value 
necessary for the evaluation of group data. Anastasi (1976) wrote that it was desirable 
for reliability coefficients to be in the range of .80s to .90s. The Cronbach alpha value for 
the item scores on the instrument was 0.88. This is certainly a most appropriate 
reliability value. 

Discussion 

The findings presented in Table 3 show that the instrument was comprised of two 
factors. The first factor consisted of questions one through four on the instrument. The 
second factor was comprised of questions five through nine. The items in the first factor 
pertained to the workshop and the training materials, while the items in the second factor 
pertained to the instructors themselves. Question number ten was a summative question 
stating, “Overall this was an excellent workshop.” This item had a factor loading of .523 
on factor one and .560 on factor two. As a summative question, this factorial complexity 
for the factor loadings would be expected. The mean for the first four questions was 4. 1 8 
(between Agree[4] and Strongly Agree[5]). The mean for questions five through nine was 
4.66 (between Agree[4] and Strongly Agree[5]). The mean for all ten questions was 4.45 
(between Agree[4] and Strongly Agree[5]). 

Because of the age of the participants, most were not as computer literate as those 
individuals who were in the 20s or 30s. Instrument question number four stated, “I can 
now write estimates on the DDS Multi-line Pro.” This question had a lower rating than 
question number ten, “Overall this was an excellent workshop.” On the surface, this 
would seem to be a contradiction; however, it shows that even though the workshop was 
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perceived as very helpful and informative, the participants’ lack of computer skills 
hindered them from stating that they were competent using the software. To address this 
issue, the training sessions should be lengthened by at least one day. The first day’s 
session should be devoted to training in computer skills and the Windows’ environment. 

For general use of the instrument, one could change the wording slightly for questions 
three and four for the specific workshop situation. Question three could be reworded to 
say, “The workshop has helped me understand [what has been presented].” Question 
four could be reworded as, “I can now [do a specific behavior or task].” These slight 
modifications will make the questions generic in application and use for various 
assessment environments. Question 10 should remain as it is currently stated. 

Conclusion 

The study focused on a psychometric analysis of a 10-item instrument designed to 
evaluate workshop training. The findings suggest that there are two dimensions for the 
instrument. The first factor was a general cluster of items that pertained to the workshop 
itself and the training materials. The second factor was a cluster of items that focused on 
the instructors. Furthermore, the authors reported the psychometric properties of the 
instrument including descriptive statistics, the correlation values for the items, the rotated 
factor loadings for the items, and the Cronbach alpha score reliability measure. The 
instrument was found to be a useful measurement scale with excellent psychometric 
properties. Suggestions were also offered for slight wording modifications for questions 
three and four to make the questions generic in application for various assessment 
environments. 
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TABLE 1 

Descriptive Statistics 



N 



1 


Minimum 


Maximum 


Mean 


Std. 

Deviation 


584 


1.00 


5.00 


4.1952 


.8197 


584 


2.00 


5.00 


4.1901 


.6804 


584 


1.00 


5.00 


4.3750 


.6099 


584 


1.00 


5.00 


3.9966 


.7748 


584 


2.00 


5.00 


4.5325 


.5639 


584 


3.00 


5.00 


4.6712 


.4985 


584 


3.00 


5.00 


4.7500 


.4527 


584 


3.00 


5.00 


4.7140 


.4816 


584 


2.00 


5.00 


4.6353 


.5325 


584 

584 


2.00 


5.00 


4.4452 


.6312 



VAR00001 
VAR00002 
VAR00003 
VAR00004 
VAR00005 
VAR00006 
VAR00007 
VAR00008 
VAR00009 
VAR00010 
Valid N (listwise) 
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VAR00009 


.269 

.395 

.390 

.288 

.682 

.672 

.589 

.710 

1.000 

.627 




00 

o 

o 

o 

o 

K 
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.272 

.334 

.377 

.195 

.637 

.636 

.757 

1.000 

.710 

.465 




VAR00007 


.233 

.299 

.353 

.203 

.543 

.623 

1.000 

.757 

.589 

.402 




VAR00006 


.220 

.336 

.361 

.219 

.661 

1.000 

.623 

.636 

.672 

.493 




VAR00005 


.309 

.433 

.436 

.338 

1.000 

.661 

.543 

.637 

.682 

.586 




VAR00004 


.258 

.528 

.511 

1.000 

.338 

.219 

.203 

.195 

.288 

.431 




VAR00003 


.327 

.605 

1.000 

.511 

.436 

.361 

.353 

.377 

.390 

.461 




VAR00002 


.376 

1.000 

.605 

.528 

.433 

.336 

.299 

.334 

.395 

.486 




VAR00001 


1.000 
.376 
.327 
.258 
. .309 
.220 
.233 
.272 
.269 
.336 




Correlation VAR0000 1 

VAR00002 
VAR00003 
VAR00004 
VAR00005 
VAR00006 
VAR00007 
VAR00008 
VAR00009 
VAR00010 



LO 

vH 



Table 3 

Rotated Component Matrix 3 





Component 


1 


2 


VAR00001 


.181 


.535 


VAR00002 


.214 


.806 


VAR00003 


.263 


.755 


VAR00004 


5.694E-02 


.798 


VAR00005 


.744 


.373 


VAR00006 


.823 


.184 


VAR00007 


.821 


.126 


VAR00008 


.872 


.158 


VAR00009 


.816 


.288 


VAR00010 


.523 


.560 



Extraction Method: Principal Component Analysis. 
Rotation Method: Varimax with Kaiser Normalization. 

a. Rotation converged in 3 iterations. 
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SA for Strongly Agree 
A for Agree 
N for Neutral 
D for Disagree or 
SD for Strongly Disagree 

Write the response for each item that best describes your thoughts about the training workshop. 

Date: / / Location (City/State): 

Instructor’s Name: 

Item 

1. The training room was comfortable 

2. The training materials were easy to follow and understand 

3. The workshop has helped me understand die information flow within the software 

4. I can now write estimates on die DDS Multi-line Pro™ 

5. The instructor(s) was well organized 

6. The instructor(s) was friendly and easy to ksten to 

7. The instructor(s) was knowledgeable about die software 

8. The instructor(s) answered questions quickly and knowledgeably 

9. Overall, die instructor^) did an excellent job 

10. Overall diis was an excellent workshop 

Please share any comments that will help us improve die quality of our training sessions. (We really 
want to know!) 

Comments: 
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